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ABSTRACT 

This  paper  presents  a  problem  of  optimal  flow  control  for  discrete- time  M\M\l  queues, 
where  the  decision-maker  seeks  to  maximize  the  throughput  subject  to  a  bound  on  the  av¬ 
erage  queue  size.  The  problem  is  cast  as  a  constrained  Markov  decision  process  and  solved 
via  Lagrangian  arguments.  The  optimal  strategy  is  shown  to  be  a  threshold  policy  which 
saturates  the  constraint.  The  method  of  analysis  proceeds  through  the  discounted  version  of 
the  Lagrangian  problems  whose  value  functions  are  shown  to  be  integer-concave.  Dynamic 
Programming  and  stochastic  comparison  ideas  constitute  the  main  ingredients  of  the  solution. 
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1.  Introduction 

Consider  a  synchronous  communication  channel  between  two  entities,  a  transmitter  and  a 
receiver,  which  are  both  equipped  with  buffers  of  infinite  capacity.  Information  is  formatted  in 
packets  and  time  is  slotted  so  that  the  duration  of  a  time  slot  coincides  with  the  transmission 
time  of  a  packet.  Packet  transmissions  are  initiated  at  the  beginning  of  a  slot.  The  channel 
is  assumed  noisy  in  that  a  packet  transmission  may  not  be  successful  with  probability  1  —  n 
in  which  case  retransmission  is  attempted  in  the  next  slot.  This  scenario  is  repeated  until 
successful  transmission  occurs,  at  which  time  the  packet  is  deleted  from  the  transmitter’s 
buffer.  Packets  arrive  at  the  transmitter  one  at  a  time  according  to  a  Bernoulli  sequence  with 
rate  A,  i.e.,  A  is  the  probability  that  a  packet  will  arrive  in  any  time  slot,  and  the  transmission 
failures  are  assumed  independent  from  slot  to  slot,  and  independent  of  the  arrival  process. 

As  this  system  may  experience  congestion,  it  is  desirable  to  take  certain  actions  in  order  to 
guarantee  an  expected  performance  level.  One  possible  approach  consists  in  restricting  access 
to  the  communication  system,  i.e.,  new  packets  which  are  about  to  enter  the  transmitter’s 
buffer  may  be  denied  entrance  on  the  basis  of  information  reflecting  system  congestion.  This 
is  often  referred  to  as  flow  control  and  should  be  done  on  the  basis  of  some  performance 
criterion  [3].  Here,  an  approach  similar  to  the  one  of  Lazar  [5]  is  adopted  in  that  a  flow 
control  strategy  is  sought  that  maximizes  the  channel  throughput  subject  to  a  constraint  on 
the  long-run  average  number  of  packets  in  the  system. 

Under  the  statistical  assumptions  given  earlier,  the  uncontrolled  system  can  be  modelled 
as  a  discrete-time  M\M\l  queue,  and  the  problem  of  finding  good  flow  control  schemes  can 
be  cast  as  a  Markov  decision  process  (MDP)  with  constraint.  Analysis  shows  that  this  con¬ 
strained  flow  control  problem  admits  a  solution  within  the  class  of  threshold  policies  which  are 
parametrized  by  an  integer-valued  threshold  level  L  (=  0, 1, . . .)  and  an  acceptance  probability 
V  (0  <  it  <  1).  A  threshold  policy  (L,  77)  has  a  simple  structure  in  that  at  the  beginning  of  each 
time  slot,  a  new  packet  is  accepted  (resp.  rejected)  if  the  buffer  content  is  strictly  below  L 
(resp.  above  L),  while  if  there  are  exactly  L  packets  in  the  buffer,  this  new  packet  is  accepted 
(resp.  rejected)  with  probability  r?  (resp.  1  —  rj). 

Throughout  the  years,  various  problems  of  flow  control  (or  control  of  arrivals)  have  been 
studied  in  the  context  of  simple  queueing  systems,  and  a  good  discussion  of  such  work  can 
be  found  in  the  survey  paper  of  Stidham  [13].  It  should  be  pointed  out  that  previous  papers 
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dealt  exclusively  with  continuous-time  models,  and  that  the  concavity  of  the  value  function 
for  the  single  node  situation  could  be  obtained  fairly  easily  through  standard  arguments. 
Here,  establishing  the  concavity  of  the  value  functions  of  interest  turns  out  to  be  a  much 
more  cumbersome  task  and  constitutes  the  key  technical  contribution  of  this  paper.  This 
difficulty  can  possibly  be  explained  by  the  fact  that  multiple  transitions  can  be  realized, 
a  phenomenon  which  precludes  use  of  the  homogeneization  technique  for  the  discrete-time 
situation  [4].  Dynamic  Programming  and  stochastic  comparison  ideas  constitute  the  main 
technical  ingredients  of  the  solution. 

Amongst  the  models  covered  in  Stidham’s  survey  paper,  only  the  work  of  Lazar  [5]  for¬ 
mulates  the  problem  as  a  constrained  problem.  However,  the  approach  taken  here  is  different 
from  the  one  used  by  Lazar  in  that  he  considers  a  closed  system  from  the  onset  with  a  fixed 
number  of  packets,  while  the  work  discussed  here  assumes  an  open  system.  Of  course,  both 
approaches  lead  to  similar  results,  as  expected. 

The  paper  is  organized  as  follows.  The  model  is  described  in  Section  2  and  the  constrained 
flow  control  problem  is  posed  in  Section  3,  where  the  optimality  results  are  summarized  and  the 
necessary  Lagrangian  are  briefly  outlined.  Section  4  is  devoted  to  the  study  of  the  discounted 
version  of  the  Lagrangian  problems,  for  which  threshold  policies  are  identified  to  be  optimal. 
The  properties  of  threshold  policies  are  discussed  in  Section  5,  and  used  in  Section  6  to  find 
the  solution  to  the  long-run  version  of  the  Lagrangian  problems.  A  useful  comparison  result 
is  given  in  the  Appendix. 

A  word  on  the  notation:  The  set  of  real  numbers  is  denoted  by  2R,  while  IN  denotes  the 
set  of  all  non-negative  integers.  For  any  x  in  2R,  it  is  convenient  to  pose  x  =  1  —  x,  and  the 
characteristic  function  of  any  set  E  is  denoted  simply  by  1[2?]. 

2.  Model 

In  order  to  formally  define  a  flow  control  model  for  discrete-time  M\M\l  systems,  start 
with  the  sample  space  fi:=  IN  x  ({0, l}3)°°,  and  recursively  define  the  information  spaces 
{ Mn }g°  by  JHq:  =  IN  and  lHn+i  :  =  IHn  x  {0,  l}3  for  all  n  =  0, 1,  — 

An  element  oj  of  fi  is  viewed  as  a  sequence  (x,  ojo,  o>i, . . .)  with  x  in  IN  and  wn  in  {0,  l}3 
for  all  n  =  0, 1, . . ..  Each  block  component  wn  is  written  in  the  form  (un,  a„,  6n),  with  un, 
an  and  bn  being  all  elements  in  {0,  l}.  An  element  hn  in  Hn  is  uniquely  associated  with  the 
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sample  u  by  hn :  =  (x,u0,  . . . ,  wn_i)  with  ho :  =  x. 

Let  the  sample  oj  =  (x,(jj0,ui,.  . .)  be  realized.  The  initial  queue  size  is  set  at  x.  During 
each  time  slot  [n,  n+  1),  an  =  1  (resp.  an=0)  indicates  that  a  customer  (resp.  no  customer) 
has  arrived  into  the  queue,  bn  =  1  (resp.  bn= 0)  encodes  a  successful  (resp.  unsuccessful) 
completion  of  service  in  that  slot,  whereas  control  action  un  is  selected  at  the  beginning  of 
the  time  slot  [re,  rc+l),  with  un=l  (resp.  un=0)  for  admitting  (resp.  rejecting)  the  incoming 
customer  during  that  slot.  If  xn  denotes  the  queue  size  at  the  beginning  of  the  slot  [n,  n+l), 
its  successive  values  are  determined  through  the  recursion 

xn+i  =  xn  +  unan  -  l[xn  #  0]6n  »  =  0,1,... (2.1) 

with  Xq-.=  x. 

The  coordinate  mappings  S,  {£/(») }§°,  {A(n.)}§°  and  {B(n)}o°  are  defined  on  the  sam¬ 
ple  space  0  by  posing  S(u>):=  »,  £/(n,u/):  =  unt  A(n,w)  :  =  an  and  B(n,u>):=  bn  for  n  = 
0,1,...  and  w  in  O,  with  the  information  mappings  {^(n)}^  being  given  by  H(n, w):  = 
(a;,a;o,a;i, . . .  ,wn_i):  =  hn. 

For  each  n  =  0, 1, . . .,  let  JFn  be  the  <r-field  generated  by  the  mapping  H{n )  on  the  sample 
space  fl.  Clearly,  lFn  C  IFn+1,  and  with  standard  notation,  2F:  =  V^L 0IFn  is  simply  the  a- 
field  on  n  generated  by  the  mappings  3  and  (f/(n),  A(n),  JB(n)}o°.  Thus,  on  the  space  (D,iF), 
the  mappings  3,  {t/(n)}o°,  {-A(n)}o°,  {B(n)}g°  and  {^(n.)}^  are  all  random  variables  (RV) 
taking  values  in  W,  {0, 1},  {0, 1},  {0, 1}  and  lHn,  respectively.  The  queue  sizes  {X(n)}o°  are 
W-valued  RV’s  recursively  defined  by 

X{n  +  1)  =  X(n)  +  U{n)A{n)  -  l[X(n)  ^  0]B(n)  n  =  0, 1, . . .  (2.2) 

with  X(0):  =  5.  Each  RY  X(n)  is  clearly  2Fn-measurable. 

Since  randomization  is  allowed,  an  admissible  policy  7r  is  defined  as  any  collection  {^n}o° 
of  mappings  7rn:  IHn  — *  [0, 1],  with  the  interpretation  that  the  potential  arrival  during  the  slot 
[n, n  +  l)  is  admitted  (resp.  rejected)  with  probability  n n(fi„)  (resp.  1  —  n n(hn))  whenever 
the  information  hn  is  available  to  the  decision-maker.  In  the  sequel,  denote  the  collection  of 
all  such  admissible  policies  by  P . 

Let  q{*)  be  a  probability  distribution  on  IN,  and  let  A  and  fj,  be  fixed  constants  in  (0, 1). 
Given  any  policy  7T  in  P,  there  exists  an  unique  probability  measure  P *  on  IF,  with  expectation 
operator  Ev,  satisfying  the  requirements  (R1)-(R3)  below,  where 


4 


(Rl):  For  all  x  in  JN, 

P*[S  =  x]:=  q(x), 

(R2):  For  all  a  and  b  in  {0, 1}, 

P*[A{n)  =  a,  B(n)  =  b\JFn  V  <r{17(n)}]:  =  (aA  +  a\)(bp  +  bp.)  n  =  0, 1, . . . 

(R3): 

P*\U{n)  =  l|iPn]:  =  7rn(P(n)).  n  =  0, 1, . . . 

This  notation  is  specialized  to  P*  and  E*,  respectively,  when  g(«)  is  the  point  mass  distribution 
at  x  in  TN\  it  is  plain  that  Pv[A\X(0)  =  x\  =  P*[A]  for  every  A  in  IF.  It  readily  follows  from 
(Rl)-(R3)  that  under  each  probability  measure  Pr, 

(PI):  The  ZV-valued  RV  E  is  independent  of  the  sequences  of  RV’s  {A(n)}o°  and  {P(n)}g°, 
(P2):  The  sequences  {A(n)}g°  and  {P(n)}g°  of  {0,  l}-valued  RV’s  are  mutually  indepen¬ 
dent  Bernoulli  sequences  with  parameters  A  and  p,  respectively,  and 
(P3):  The  transition  probabilities  take  the  form 

P*[X{n  +  1)  =  y|lF„]  =  p[X(n),y;  7rn(P(n))]  n  =  0, 1, . . .  (2.3) 

where 

p[x,y]ri]:=riQ1(x,y)  +  fjQ0(x,y)  (2.4) 

with 

Q'(x,y):=  P*\x  +  t’A(n)  -  l(z  ^  0)P(n)  =  y],  *  =  0, 1  (2.5) 

for  all  x  and  y  in  IN,  and  all  rj  in  [0,1]. 

The  right-hand  sides  of  (2.5)  depend  neither  on  n  nor  on  the  policy  n  owing  to  the  assumptions 
(R1)-(R3)  made  earlier.  Throughout  this  paper,  the  RV  S  is  assumed  to  have  finite  mean,  i.e., 

£r=0  *?(*)  <  °°* 

Several  subclasses  of  policies  in  P  will  be  of  interest  in  the  sequel. 

A  policy  jt  in  P  is  said  to  be  a  Markov  policy  if  there  exists  a  family  {yn}§°  °f  mappings 
gn :  IN  ->  [0, 1]  such  that  7rn(fi'(n))  =  gn(X(n))  P"  -  a.s.  for  all  n  =  0, 1, . . .  In  the  event  gn=g 
for  all  n  —  0,1,...,  the  Markov  policy  ir  is  called  stationary  and  can  be  identified  with  the 
mapping  g  itself. 
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A  policy  7T  in  P  is  said  to  be  a  pure  (or  non-randomized)  policy  if  there  exists  a  family 
{/n}o°  of  mappings  -»  {0, 1}  such  that  nn(H(n))  =  fn(H(n))  P *  -  a.s.  for  all  n  = 

0, 1, . . .  A  pure  Markov  stationary  policy  tt  is  thus  fully  characterized  by  a  single  mapping 

f:N-+{  0,1}- 

A  stationary  policy  g  is  said  to  be  of  threshold  type  if  there  exists  a  pair  ( L,r ]),  with  L  in 
IN  and  i]  in  [0,  l],  such  that 

(1  if  x  <  L] 

g{x)  =  <  7/  if  x  =  L\  (2.6) 

l  0  if  x  >  L. 

Such  a  threshold  policy  is  denoted  by  and  by  extension,  the  Markov  stationary  policy 

g  that  admits  every  single  customer,  i.e.,  g(x)  =  1  for  all  x  in  IN,  is  conveniently  denoted  by 
(oo,i). 

3.  The  optimal  control  problems 

For  any  admissible  policy  n  in  P,  pose 

#  0]  (3.1) 

n+1  ^ 

and 

N(ir):  =  \\msupn— ErJ2x(t).  (3.2) 

n  +  1  t=o 

These  quantities  T(n)  and  N(n)  are  readily  interpreted  as  the  throughput  and  the  long-run 
average  queue  size,  respectively,  when  the  policy  7 r  is  used. 

Given  V  >  0,  the  problem  ( Py )  of  interest  is  the  constrained  optimization  problem 

(Pv)’  Maximize  T( tt)  over  Py 


where 

Pv  :=  {tt  G  P  :  N{ir)  <  V}.  (3.3) 

If  the  constraint  is  satisfied  when  admitting  every  single  customer,  i.e.,  iV((oo,l))  <  V, 
then  Py  =  P  and  the  constrained  optimization  problem  (Pv)  reduces  to  an  unconstrained 
optimization  problem  with  trivial  solution  (oo,  1)  as  shown  in  [6,  Thm.  3.1]  by  simple  stochastic 
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comparison  arguments.  On  the  other  hand,  if  N((oo,  l))  >  V,  the  solution  to  the  constrained 
problem  ( Pv )  is  no  longer  trivial  and  it  is  the  main  objective  of  this  paper  to  identify  its 
structure.  The  main  result  is  summarized  in 

Theorem  3.1  If  N((oo,  l))  >  V,  then  there  exists  a  threshold  policy  (L*,tj*)  which  solves 
problem  (Pv)  with  N((L*,rj*))  =  V. 

The  proof  of  Theorem  3.1  is  outlined  in  Section  6.  The  solution  method  for  these  con¬ 
strained  optimization  problems  uses  Lagrangian  arguments  similar  to  the  ones  given  in  [2,10]. 
Here,  the  appropriate  Lagrangian  functional  is  defined  for  any  admissible  policy  tt  in  P  to  be 

=  lim inf n— —E1' Y2fil[X(t)  ^  0]  -~tX(t)  (3.4) 

•*»  "T"  l  .  _ 

t—0 

with  *7  >  0  denoting  the  Lagrange  multiplier.  The  corresponding  Lagrangian  problem  (LP'1) 
is  then  the  unconstrained  problem 

( LP 'r):  Maximize  J',(7r)  over  P. 

Under  (P1)-(P3),  each  unconstrained  problem  (LP'1),i  >  0,  can  be  viewed  as  a  Markov 
decision  problem  under  the  long-run  average  cost  criterion,  with  state  process  {X(n)}g°,  cost 
per  stage  c1:  IN  — >  1R  given  by 


e1(x):=  fil[x  ^  0]  -  7x  (3.5) 

for  all  x  in  IN,  and  information  pattern  {H(n)}g°.  Although  the  information  pattern  H(n)  is 
richer  than  the  standard  state  feedback  information  pattern  {H, U(k), X(k  +  1),0  <  k  <  n}, 
it  is  easy  to  see  that  the  value  functions  for  the  corresponding  discounted  problems  coincide. 

If  7  ^  fj,,  then  c"1  (x)  ^  0  for  all  x  in  IN,  and  an  elementary  argument  now  shows  that 
^((0,0))  =  0  >  7^(7 r)  for  all  x  in  P,  whence  the  threshold  policy  (0,0)  trivially  solves  the 
Lagrangian  problem  (LP'1).  Therefore,  only  the  case  7  <  p,  needs  to  be  investigated  and  this 
is  done  in  the  next  section  by  considering  the  appropriate  discounted  problems. 

4.  The  discounted  problems 
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Let  ii  >  7  >  0  and  0  <  0  <  1  be  held  fixed  throughout  this  section.  The  expected 
^•discounted  Lagrangian  cost  Jp(n)  associated  with  an  admissible  policy  7r  in  P  is  defined  by 

(4.1) 

t= 0 


and  the  corresponding  discounted  optimization  problem  ( LP ^ )  is  then  simply 


(LPp):  Maximize  Jg(7r)  over  P . 


Since  at  most  one  arrival  can  be  admitted  in  each  time  slot,  the  pathwise  bound  X  (n.)  <  5  +  n 
holds  for  all  n  =  0, 1, . . .  and  yields  the  estimate 


IWI  < 


H  +  ^En[  5]  -i/3 

1-0  +(1-/?)2 


<  oo. 


(4.2) 


The  bound  (4.2)  being  independent  of  the  policy  n  in  P,  the  quantity  Jj  (7r)  is  thus  well-defined 
and  uniformly  bounded  over  P. 

As  customary  with  the  Dynamic  Programming  methodology,  the  0- discounted  cost-to-go 
associated  with  any  policy  7r  in  P  is  the  mapping  J^,7r:  IN  — >  JR  defined  by 

JJ-'(*):=£I[fy«’(X(t))]  (4.3a) 

t= 0 


for  all  x  in  IN,  while  the  corresponding  value  function  V^-.IN  — *  IR  is  given  by 


Vp  (x):  =  sup  J^(x). 

ir&P 


(4.36) 


Let  the  RV’s  A  and  B  be  generic  elements  in  {A(n)}o°  and  {5(n)}g°,  respectively,  and 
for  all  x  in  IN,  define  the  IN-v alued  RV’s  A°(i)  and  Al{ x)  by 

Ax{x )  =  x  +  t A  —  l[x  7^  0] JB,  i  =  0, 1.  (4*4) 

For  any  mapping  f:  IN  —*  IR,  define  the  mapping  Tj/j  f:  IN  — ►  2R  by 

PJ/)(*)  =  C(x)  +  PmaxlnEVlA^x)))  +  ?£[/(A°(*))l}  (4.5) 
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for  all  x  in  IN.  Here,  for  each  i  =  0,1,  £7[/(yl*(s))]:  =  E*  [f  (A1  (x))]  for  all  tt  in  P  owing  to 
(2.3)-(2.5),  with 

E\f(A1(x))\  =  l  +  ^(°)  _  if  35  =  05 

^  \  Xfif(x  -  1)  +  (Xfi  +  Xfi)f(x)  +  Xp,f(x  +  1)  if  x>l 

and 

^/M  =  {^_1)+p/w  l  III 

For  future  reference,  for  any  mapping  /:  IN  —>  1R,  pose 

„/WaBJ/( i)-/(0)  if  *  =  o; 

'  \m(/(x)  -  /(x  -  1))  +  £(/(x  +  1)  -  /(*))  if  *>1 

and  observe  that 

ElfiA'ix))]  -  E[f{A°{x))]  =  X  V  f(x) 

for  all  x  in  IN. 

The  backward  induction  of  Dynamic  Programming  produces  the  sequence  {Vp}  o°  of  map¬ 
pings  Va:  IN  — *  JR  through  the  recursion 

V+1=rW  »  =  0,1,...  (4.9) 

with  Vp\  =■  c7.  The  cost  e7  being  bounded  above  by  n  —  7,  the  discounted  problem  ( LPp ) 
is  covered  by  Assumption  P  of  Bertsekas  [1,  pp.  251].  The  following  theorem  is  now  readily 
obtained  by  specializing  results  from  Section  6.4  of  Bertsekas  [l]. 

Theorem  4.1  The  value  function  Vp  satisfies  the  Dynamic  Programming  equation 

V  =  TJVj  (4.10) 

and  is  obtained  as  the  pointwise  limit 

lim„  V?(x)  =  Vj(x)  (4.11) 

for  all  x  in  IN.  Moreover,  the  Markov  stationary  policy  g*  in  P  defined  by 

(  1  if  V^(x)  >  0; 

g*(x)  =  <  arbitrary  in  [0,1]  if  s/Vp  (x)  =  0;  (4-12) 

(0  if  VV  (x)  <  0  . 


(4.6a) 

(4.65) 

(4.7) 

(4.8) 
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is  optimal  for  problem  ( LPjf ). 

The  value  iteration  method  implicit  in  Theorem  4.1  constitutes  a  powerful  tool  for  further 
characterizing  the  structure  of  the  optimal  policy.  Some  insights  are  already  obtained  through 
Lemma  4.2  below.  Pose  L'1:  =  max{Z  €  IN:  p  —  ql  >  0},  and  observe  that  L'1  >  1. 

Lemma  4.2  For  the  discounted  problem  (LPjf),  an  optimal  Markov  stationary  policy  g*  can 
always  be  chosen  so  that  g*(x)  =  0  for  all  x  >  L7. 

Proof.  Define  the  IFn- stopping  time  r  by 


f  inf{  k  >  0  :  X[k)  =  L7}  if  the  set  is  non-empty; 
1  oo  otherwise 


(4.13) 


with  the  obvious  interpretation  that  t  is  the  first  passage  time  into  the  state  L'1.  For  any 
admissible  policy  ir  in  P,  pose  S|( as):  =  E£[f}T]  and 


T  — 1 


(*)■--  E;[X>V'pr(())]  =  E!(£p‘  i[r  >  <]c7(jr(i))l 


t—Q 


t=0 


for  all  x  in  IN.  Since  an  optimal  stationary  policy  exists  by  Theorem  4.1,  a  simple  argument 
based  on  the  strong  Markov  property  readily  shows  that  the  value  function  Vp  satisfies  the 
relation 

Vf(x)  =  max{7^'(i)  +  V„’(L7)BJM}  (4.14) 

for  all  x  in  IN,  with  the  maximization  being  taken  over  all  stationary  policies  g  in  P. 

Now,  from  any  arbitrary  stationary  policy  g  in  P,  construct  from  it  a  new  policy  g  which 
generates  actions  according  to 


if  0  <  x  <  L1  ; 
if  x  >  L'1. 


(4.15) 


Since  V^L'1)  >  Jp’('°’°\L'1)  >  c'1(L'*)  >  0,  Lemma  4.2  will  now  be  established  by  showing 
that  Ip’9{x)  <  Ip’3{x)  and  Bp(x)  <  B9p(x)  for  all  x  in  IN. 

If  0  <  x  <  L7,  the  probability  measures  P|  and  P|  coincide  on  the  a-field  lFr,  whence 
IJ'i(x)  =  Pp’~9{x)  and  B9p{x)  =  P|(x).  For  x  >  L7,  pose  Z(t)  =  l[r  >  t)X(t)  for  all  t  =  0, 1,. . .. 
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The  reader  will  now  check  from  the  very  definition  of  g  and  from  the  proof  of  Theorem  A.2  in 
the  Appendix  that 

mt)}o,Pi)  <st  {{Z{t))o>Pi)'  (4.16) 

Moreover,  for  t  =  0,1,...,  the  relations  l[r  >  t]  =  f{Z(l), . . . ,  Z{t))  and  l[r  >  = 

—h(Z(l),...,Z(t))  take  place  both  P|- a.s.  and  Pj-a.s.,  with  monotone  non- deer  easing  map¬ 
pings  /,  hilN*  —*■  1R.  It  is  now  immediate  from  (4.16)  that  (r,  P|)  <3t  (r,  P|),  or  equivalently, 
that  (/?T,P.j?)  <at  (/3T,P|)  and  that 

(l[r  >  ty{X{t)),Pi)  <3t  (l[r  >  t]c*(X(i)),Pf) 

for  all  t  =  0,1,...  The  inequalities  Bp(x)  <  B^(x)  and  Ij'°{x)  <  Ij’^(x)  are  now  readily 
obtained.  n 

By  virtue  of  Lemma  4.2,  the  Dynamic  Programming  equation  (4.10)  reduces  to 

V?W  =  S  (*)  +  f>E\V?(A°(x)))  (4.17) 

for  all  x  >  L">,  and  easy  algebraic  manipulations  yield 

W  +  1)  -  V/(L’)  <  0.  (4.18) 

The  value  iteration  method  of  Theorem  4.1  is  now  used  to  establish  the  integer -concavity 
of  the  value  function  Vp  by  showing  the  concavity  of  each  one  of  the  iterates  {V^Iq5  given  by 
(4.9).  Surprisingly  enough,  this  turns  out  to  be  a  non- trivial  task  as  several  situations  need  to 
be  discussed  separately.  The  difficulty  seems  to  stem  from  the  fact  that  multiple  transitions 
are  possible  here  owing  to  the  discrete  nature  of  the  time  parameter.  This  is  in  contrast  with 
the  continuous-time  version  of  this  problem  for  which  concavity  of  the  value  function  is  more 
readily  obtained  through  some  of  the  arguments  of  [13]. 

The  next  result  shows  in  what  sense  integer-concavity  is  preserved  under  the  backward 
induction  of  Dynamic  Programming.  It  will  be  convenient  to  say  that  a  mapping  f'.JN^JR 
satisfies  the  property  (At),  i  —  1, . . .  ,4,  if 

(Al):  /  is  integer-concave  with  0  <  /(i)  —  /(°)  <  m  —  nr, 

(A2):  /  is  integer-concave  with  fi  —  7  <  m  -  m  <  1, 

(A3):  /  is  integer-concave  with  /( 2)  —  /( 1)  <  —7, 


11 


(A4):  /  is  integer-concave  with  /(2)  —  /(l)  >  —7. 

Theorem  4.3  (i)  Suppose  A  +  p  <  1.  7/  /  satisfies  (A2),  so  does  Tpf.  (ii)  Suppose  \  +  p  >  1 
and  p2  <7.  ///  satisfies  (Al)  and  (A3),  so  does  Tpf.  (iii)  Suppose  X  +  p  >  1  and  ^  <  A  <  p. 
If  f  satisfies  (A2)  and  (A4),  so  does  Tpf.  (iv)  Suppose  A  -f-  p  >  1  and  A  <  ^  <  p.  If  f 
satisfies  (Al)  with  V/(l)  <  0,  then  Tjff  satisfies  (Al)  and  (A3).  If  f  satisfies  (Al)  and  (A3) 
with  V/(l)  >  0)  then  Tpf  satisfies  (Al)  and  (A4).  If  f  satisfies  either  (Al)  and  (A4)  with 
V/(l)  >  or  (A2)  and  (A4)  ( whence  V/(l)  >  0  necessarily ),  then  Tpf  satisfies  either  (Al) 
and  (A4),  or  (A2)  and  (A4). 

A  complete  discussion  of  this  key  result  can  be  found  in  [6,  Appendix  II].  It  is  noteworthy 
that  concavity  alone  does  not  propagate  under  the  induction  argument  and  that  additional 
growth  conditions  are  needed. 

Theorem  4.4  The  value  function  Vp  is  integer-concave,  with 

W  =  v?(°)  <  v; (1)  aud  v;(L-'  +  x)<v;(i7  (4.19) 

Proof.  The  argument  is  standard  and  inductively  uses  Theorem  4.3  on  the  successive  iterates 
{^/?}o°*  This  is  made  possible  by  observing  that  the  0th  iterate  Vp  is  the  concave  mapping  c1 
which  satisfies  c^l)  —  c'7(0)  =  p  —  7  and  c'7(2)  —  0^(1)  =  —7.  The  reader  will  now  check  that 
each  one  of  the  four  situations  discussed  in  Theorem  4.3  applies  to  yield  the  integer-concavity 

of  Vp  with  0  <  V^(l)  —  V^O)  <  1  for  all  n  =  1,2, _ 

In  the  limit,  by  Theorem  4.1,  the  value  function  Vp  is  thus  integer-concave  with  0  < 
Vp(  1)  —  Vy(0)  <  1.  Consequently,  V^/(0)  >  0  and  <7*(0)  =  1  is  an  optimal  action  by  (4.12), 
whence  Vp  (0)  =  fl[XVp  (l)  +  A  Vp  (0)]  by  virtue  of  the  Dynamic  Programming  equation  (4.10). 

The  first  part  of  (4.19)  now  follows  from  the  fact  that  Vp  ( 1)  >  Jp’^°’°\  1)  >  p  —  7  >  0,  while 
the  second  part  is  nothing  but  (4.18).  □ 

Theorem  4.5  For  every  0  <  (3  <  1,  there  exists  a  threshold  policy  {L'p,  l)  which  solves  problem 
(LPp),  where  the  optimal  threshold  value  L'p  satisfies  the  relation  0  <  L}  <  L\ 

Proof.  Integer-concavity  of  implies  S7Vp  to  be  monotone  decreasing,  whence  the  quantity 
syVp  changes  sign  at  most  once,  from  positive  to  negative,  as  a  consequence  of  (4.19).  There¬ 
fore,  there  exists  a  level  L'p,  with  0  <  L'p  <  L'1,  such  that  \jVp  (2)  >  0  for  0  <  x  <  Lj  and 
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S7Vp(x)  <  0  for  all  x  >  Lj.  The  threshold  policy  (Lp,  1)  is  then  clearly  optimal  by  Theorem 

4.1.  □ 

5.  Properties  of  threshold  policies 

For  each  threshold  policy  ( L,i j)  with  L  in  IN  and  0  <  tj  <  1,  the  sequence  {X(n)}o°  is 
a  time- homogeneous  Markov  chain  with  state  space  IN  under  the  probability  measure  p(L,rl\ 
This  chain  has  a  single  ergodic  set,  namely,  {0, 1, ...,L}  if  v  =  0  or  {0, 1, ... ,  L+l}  if  0  <  tj  <  1, 
with  all  the  other  states  being  transient.  Consequently,  the  Markov  chain  {X(n)  admits 
under  a  unique  invariant  measure,  which  is  denoted  by  IP (L,T>)  with  corresponding 

expectation  operator  JE^L,7,\  As  pointed  out  in  [6,  Sec.  5],  routine  calculations  yield  closed- 
form  expressions  for  this  invariant  measure  1P(L>,I\ 

Let  X  denote  a  generic  W-valued  random  variable.  For  any  mapping  d  :  IN  —*  IR,  the 
quantity  1E^L,V^ d(X)  is  always  finite  since  has  finite  support.  Furthermore,  the  first 

passage  time  to  the  set  of  ergodic  states  being  a.s.  finite  under  the  threshold  policy  (L,r/),  the 
queue  sizes  {X(n)}g°  satisfy  the  inequality 

X(n)  <  E  V  (L  +  1)  p(L’r>'>  -  a.s.  n  =  0,1,...  (5.1) 

The  next  result  is  now  immediate  from  standard  properties  of  Markov  chains. 

Lemma  5.1  If  the  mapping  d:lN  — >  2R  is  monotone  and  the  RV  d( E)  is  integrable,  then  the 
RV’s  {d(X(n.))}o°  are  uniformly  integrable  under  p(L’ri))  and  the  convergence 

limn — dW))  =  E{L,n)d{X)  P^’")  -  a.s.  (5.2) 

n  +  1  t=o 

takes  place,  independently  of  the  initial  distribution,  both  P(L,r>}  —  a.s.  and  in  L1{U,1F,P^l^). 

The  next  lemma  will  be  useful  in  proving  the  main  result  of  Section  6.  The  obtained 
results  are  also  applicable  to  various  situations  discussed  in  the  companion  papers  [7,8]. 

Lemma  5.2  For  any  mapping  d:JN-+lR  and  any  threshold  policy  ( L,i j),  there  always  exist 
a  scalar  J  and  a  mapping  h:  IN  — +  IR  such  that 

h(x)  +  J  =  d(x)  +  (L,rj)(x)E[h(A1(x))}  +  (L,t])(x)E[h(A°(x))}  (5.3) 
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for  all  x  in  IN.  The  quantity  J  is  given  by 

j  =  1!m"rxTi^”’)E  W))]  =  (5.4) 

t=0 

whereas  the  mapping  h:  IN  — >  1R  is  unique  up  to  an  additive  constant,  and  under  the  constraint 
h(L)  =  0,  is  given  by 

h(x)  =  <W))1  -  (5.5) 

t=0 

for  all  x  in  IN.  Here  r  is  the  JFn-stopping  time  defined  by  (4-18)  but  with  L  instead  of  L1. 

The  proof  of  Lemma  5.2  is  by  now  standard  and  is  omitted  for  sake  of  brevity.  The 
interested  reader  is  invited  to  consult  the  monograph  by  Ross  [11]  or  the  work  by  Shwartz 
and  Makowski  [12]  for  a  typical  discussion.  Crucial  to  the  argument  is  the  fact  that  the  chain 
{X(n)}o°  reduces  to  a  finite-state  Markov  chain  under  PxL,v ^  for  every  x  in  IN. 

It  is  clear  from  (5.4)  and  Lemma  5.1  that  the  equality  J'7((L,r;))  =  T((L,rj))  —^N((L,ri)) 
holds,  with  T((L,rj))  =  filP^^X  ^  0]  and  N((L,i 7))  =  lE^’^X  for  every  L  in  IN  and 
0  <  77  <  1-  The  following  properties  are  obtained  by  routine  inspection. 

Lemma  5.3  For  every  L  in  IN,  the  mappings  7  — ►  T((L,r]))  and  rj  — >■  N((L,r)))  are  continu¬ 
ously  differentiable  and  strictly  monotone  increasing  on  the  interval  [0, 1],  whence  the  quantities 
T((L,0))  and  N((L, 0))  increase  as  L  increases. 

Lemma  5.4  For  each  7  >0,  the  mapping  L  — >  Jnr((L,  0))  is  unimodal,  with  the  global  maxi¬ 
mum  being  achieved  at  at  most  two  adjacent  levels. 

6.  The  long-run  average  problems 

The  value  L'1  of  Theorem  4.5  is  independent  of  the  policy  n  and  of  the  discount  factor  /?, 
and  this  makes  it  possible  to  solve  the  long-run  average  problem  ( LP 7)  by  standard  Tauberian 
arguments  applied  to  the  discounted  problem  ( LPp ).  This  result,  proved  in  [6,  Thm.  6.1],  is 
now  summarized. 

Theorem  6.1  For  each  7  >  0,  there  always  exists  a  threshold  policy  (L*,7*),  with  0  < 
L*  <  L1 ,  which  solves  the  long-run  average  problem  (LP'1)  and  yields  the  optimal  cost  J7  as 
J7  =  lE^L'i’r}'^c1(X).  If  p  —  7  <  0,  i.e.,  L7  =  0,  then  necessarily  77*  =  0  and  J 7  =  0,  while  if 
(i  —  7  >  0,  i.e.,  L7  >  0,  then  rj*  can  always  be  chosen  to  be  1. 
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For  each  7  >  0,  the  search  for  an  optimal  policy  can  therefore  be  restricted  to  the  class 


of  all  pure  threshold  policies  with  threshold  below  level  L'1  +  1,  and  the  optimal  cost  J1  of 
problem  ( LP 7)  can  simply  be  written  as 


J1  =  max  J^UL,  0))  =  max  lE^’^c^fX). 

0<L<L-I+1  vv  "  Letf  v  ' 


(6.1) 


The  results  on  threshold  policies  obtained  in  Section  5  can  be  used  to  identify  the  optimal 
threshold  policy  through  (6.1).  This  idea  is  now  exploited  to  produce  a  stronger  result  which 
is  essential  to  solving  the  constrained  problem  (Py). 

Theorem  6.2  For  each  threshold  value  L  in  IN,  there  always  exists  't(L)  >  0,  with  L'1^  >  L, 
so  that  any  admissible  policy  it  in  P  given  by 

(  1  if  X(n)  <  L\ 

irn(JT(n))  =  <  arbitrary  in  [0,  l]  if  X(n)  =  L;  (6.2) 

[0  if  X(n)  >  L 

solves  the  long-run  average  problem  ( LP '1^). 

The  following  result  will  be  useful  in  the  proof  of  Theorem  6.2. 

Lemma  6.3  Let  the  pair  ( h,J )  obtained  in  Lemma  5.2 .  If  the  sequence  {d(X(re))}o°  is  uni¬ 
formly  integrable  under  p(L>t>),  then  the  convergence 


limn^^jp(L’^  [/>(*(«  +  1))]  -  ^’^(S)]  j  =  0 


(6.3) 


takes  place. 

A  proof  of  Theorem  6.2.  For  each  L  in  IN,  pose 


r((M))-r((£,o)) 

W',-W((£,l))-tf((L,0))- 

It  is  plain  that  ^(L)  >  0  owing  to  Lemma  5.3.  and  that  (6.4)  is  equivalent  to 
J'tW^L, 0)).  It  then  follows  from  Lemma  5.4  and  (6.1)  that 

r^((L,  1))  =  0))  =  max  ((1,0))  =  J«L\ 


(6.4) 

J^)((L,1))  = 


(6.5) 
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and  both  policies  (L,  1)  and  (L,0)  solve  problem  (LP'1^).  To  prove  Theorem  6.2,  i.e.,  that 
any  policy  tt  of  the  form  (6.2)  is  optimal  for  problem  (LP'1^),  it  only  remains  to  show  that 
=  Note  that  the  policies  (L,  1)  and  (L,  0)  are  clearly  among  these  policies. 

Take  the  mapping  d  =  c1^  in  Lemma  5.2,  and  let  (hi,  Ji )  be  the  corresponding  solution 
to  the  Poisson  equation  (5.4)  associated  with  the  threshold  policy  (L,i),  i  =  0, 1,  when  hi(L)  — 
h0(L)  =  0.  The  relation  (6.5)  yields  J\  =  Jo  =  J^L\  whereas  direct  inspection  of  (5.5)  reveals 
hi(x)  =  h0(x)  for  all  x  ^  L  by  the  very  definition  of  the  stopping  time  r.  The  condition 
hi(L)  =  h0(L)  —  0  immediately  implies  that  hi  =  ho  =:h.  Consequently,  the  Poisson 
equations  associated  with  the  two  policies  ( L ,  1)  and  (L,  0)  must  coincide,  with  E[h(A1  (L))]  = 
E[h(A°(L))\,  and  must  be  of  the  form 

h(x)  4-  =  et^(x)  +  p(a:)JE[h(A1(x))]  -f  p(x)JE?[/i(.A0(x))]  (6-6) 

for  all  x  in  IN,  where 


f  1  if  0  <  x  <  L; 

p(x)  =  <  arbitrary  in  [0, 1]  if  x  =  L\ 
l  0  if  x  >  L. 

Under  the  policy  7r  in  P  defined  by  (6.2),  a  standard  argument  based  on  (6.6)  leads  to 
the  relation 

J1<1)  =  -tt £'X>’<10 (*M)  +  -iT{jHMX(»+ 1))1  -  E'[MS)]|  »  =  0, 1,... (6.7) 

n+ 1  tr'o  n  +  H  J 

By  the  very  form  (6.2)  assumed  for  tt  and  the  definition  of  r,  it  is  easy  to  see  that  for  any 
0  <  v  <  1,  both  probability  measures  p(L,ri)  and  P *  coincide  on  IFT  .  As  a  result, 

limn^  jiHMX(n+  1))]  -  ^*[*(2)]}  =  0  (6.8) 

upon  invoking  Lemma  6.3.  The  relation  J^L\ 7r)  =  J '’W  is  now  obtained  by  taking  the  limit 
in  (6.7)  and  making  use  of  (6.8).  □ 

A  proof  of  Theorem  3.1.  It  should  be  clear  from  Theorems  5.1  and  6.2  that  any  threshold 
policy  (L,rj),  with  rj  arbitrary  in  [0,1],  yields  (3.1)-(3.2)  as  limits  and  solves  problem  (LP1) 
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for  some  'v(L)  >  0.  As  discussed  in  [9],  in  order  to  solve  problem  (Py)>  it  only  remains  to  find 
a  threshold  policy  that  saturates  the  constraint. 

Since  N((0, 0))  =  0,  Lemma  5.3  readily  implies  the  existence  and  uniqueness  of  the  pair 
{L*,V*)  such  that  N((L* ,r)*))  =  V  if  iV((oo,  l))  >  V .  The  optimal  threshold  and  bias  values 
L*  and  tj*  are  uniquely  defined  by  solving  JE^'^X  =  V,  0  <  r\  <  1  and  L  —  0, 1, . . ..  □ 
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Appendix 

A  comparison  result 

The  notion  of  stochastic  ordering  is  useful  for  comparing  the  performance  of  various  poli¬ 
cies.  This  is  achieved  through  a  stochastic  comparison  result  for  the  underlying  queue  else 
process.  The  reader  is  invited  to  consult  the  monograph  by  Stoyan  [14]  for  further  information 
on  stochastic  orderings. 

Let  P 1  and  P2  be  two  probability  measures  defined  on  IF,  with  expectation  operators  E1 
and  E 2,  respectively.  If  Y  is  an  2Rn-valued  RV  defined  on  (ft,  IF),  then  the  RV  ( Y,P 2)  is  said 
to  be  stochastically  larger  than  (T^P1)  if  and  only  if  E1[f(Y)\  <  E2[f{Y)\  for  all  increasing 
functions  f:IRn  — >  IR  for  which  these  expectations  exist;  this  is  customarily  denoted  by 

(Y,PX)  (X,P2). 

This  notion  extends  naturally  to  sequences  of  2R-valued  RV’s  defined  on  (f l,  IF).  The 
sequence  ({Y(£)}o°j  P2)  *s  said  to  be  stochastically  larger  than  ({Y(f)}o°,  Pl)  if  and  only  if 

((Y(0),Y(l),...,Y(»)),P1)  <st  ((Y(0),Y(l),...,Y(n)),P2)  n  =  0, 1, . . .  (A.l) 

This  is  denoted  simply  by  ({Y(t)}g°,  P1)  <3t  ({Y(t)}o°, P2)  and  amounts  to 

P1[/(^(0),Y(l),...,Y(n))]<P2[/(F(0),Y(l),...,Y(n))]  n  =  0, 1, . . .  (A.2) 

for  all  increasing  functions  f:IRn+1  — >  IR  for  which  the  expectations  exist  [14,  Thm.  4.1.2, 

pp.  61]. 

While  the  relation  (A.2)  is  usually  hard  to  verify  directly  in  practice,  sufficient  conditions 
are  available  in  the  literature.  One  such  condition,  due  to  Veinott  [14,  pp.  29],  is  given  below 
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for  easy  reference.  Throughout  the  discussion,  the  IRn+1  -valued  RV  (Y(0), . . . ,  Y  (n))  and  the 
element  (yo,. . .  ,yn)  of  JRn+l  are  denoted  by  and  y(n\  respectively. 

Lemma  A.l  Let  {y(f)}o°  be  a  sequence  of  JR- valued  RV’s  on  (Cl,  IF).  If 


(X{0)>Pl)<«{Y{0),P*)  (A.Za) 

and  for  every  a  in  JR, 

Px[Y(n  +  1)  >  a\Y <n>  =  jW]  <  P2[Y(n  +  1)  >  a\Y <n>  =  y<n>]  n  =  0, 1, . . .  (A.Zb) 

whenever  x ^  <  y(n)  componentwise  in  JNn+1,  then  P1)  <3t  ({^(i)}o°5-f>2)‘ 

Here,  this  result  is  used  as  follows.  For  every  admissible  policy  n  in  P ,  introduce  the 
sequence  {jTn}o°  °f  mappings  ?rn:  jSVn+1  — *•  [0,  l]  defined  by 

=  P*\U(n)  =  l|X<n>  =  a<n>]  =  S’r[jrn(P‘(n))|<X'(n)  =  *(n)]  n  -  0, 1, , . ,  (A, 4) 

for  all  a;(n)  in  JNn+1. 

Theorem  A. 2  Consider  two  admissible  policies  it1  and  tt2  in  P.  If  the  relations 

*i(*w)  <  *;(y(”>)  »  -  o,  1, . . .  (A5«) 

hold  for  all  x^  <  y with  xn  =  yn,  and  if 

A7ri(x(n_1),  0)  <  /2  +  A/dr2  (y(n_1),  !)  n  =  0, 1, . . .  ( A.5b ) 

holds  for  all  x^-V  <  y^"1),  then  ({X (t)}$> ,  P*' )  <3t  ({X(t)}8°,  P**). 

Proof.  Since  the  probability  distribution  of  5  is  independent  of  the  policy,  the  relation  (A.3a) 
trivially  holds.  It  suffices  to  show  that  the  conditions  (A.5)  imply  (A.3b). 

Routine  calculations  first  imply  via  (2.3)-(2.5)  that  for  every  policy  tt  in  P, 

11  if  xn  >  a  +  1; 

p.  +  \fjLTn(xW)  if  xn  =  a+l; 

Xpjrn(xM)  if  xn  =  a  >  0;  n  =  0, 1, . . .  (A.6) 

\wn(xW)  if  xn  =  a  =  0; 

0,  if  0  <  xn  <  a 

for  all  a  in  TV  and  x^  in  lNn+1.  If  a  >  0,  0  <  A/dr n(x^n_1),  o)  <  p  +  A/drre(y(n_1),  a+  1)  <  1 
for  all  x(n~1)  and  y(n_1)  in  IRn,  and  (A.36)  thus  holds  whenever  x ^  <  y(n)  with  0  <  xn  <  yn 
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for  any  two  arbitrary  policies  in  P.  For  the  policies  tt1  and  jt2  considered  here,  it  is  now  plain 
that  (A5)  and  (A. 6)  combine  to  yield  ( A.3b )  whenever  x <  y(n),  and  the  conclusion  now 
follows  from  Lemma  A.l.  □ 

Theorem  A. 3  Consider  two  admissible  policies  and  ir2  in  P.  If  there  exists  a  sequence 
{/n}o°  °f  mappings  fn-HI  —■ ►  [0, 1]  such  that 

7T i(hn)  <  fn{xn)  <  7 r£(/&„)  »  =  0,  1, .  .  .  (A.7d) 

for  all  hn  in  lEn,  with 

A/„( 0)  <  Ji  +  A/i/n(l),  n  =  0, 1, . . .  (A.7b) 

then  ({*(*)  }^,P*1)  <4t  ({X(i)}g°,P*2). 

Proof.  It  is  plain  from  (A.4)  and  [A. 7)  that  for  all  x(n)  and  y(n)  in  J 2Vn+1, 7ri(z(n))  <  fn{xn), 
>  fn{yn),  A7ri(x(n_1),0)  <  A/n(0)  and^+A/i7r2(y(n“1),l)  >  /Z+A^/n(l).  Condition 
(A.5)  is  now  easily  justified,  and  the  result  follows  from  Theorem  A.2.  □ 
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